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(57) Abstract: A remote troubleshooting system that 
involves video, audio and data communications is 
disclosed. The system can be used for perfonning remote 
troubleshooting with the stq>port of distant customer 
service experts. In troubleshooting ^^ications, the video 
component of the system originating at factory premises 
is the main point of interest Eor the identification of 
problems during troubleshooting, good quality video is 
usually required. If this is not achievable using real-time 
video transmission, video clips can be transmitted instead 
and this need not be done in real-time. After the video clip 
is transmitted, means of synchronisation is incorporated 
to enable synchronised playback of die video at the 
troubleshooting site and the customer service centre, 
thus allowing a site service technician and the customer 
service expert to view identical videos simultaneously. 
In addition, visual interactivity is achieved by means 
of overlaying cursor objects, text and graphics onto the 
video. Two cursor objects, one from the site service 
technician and the other fix)m the customer service expert, 
is displayed on the video or images. The descriptions 
on the locations as well as the types of objects to be 
displayed are embedded in a set of data. Thus, the system 
allows a basis for further troubleshooting sessions and 
discussions with the aid of video and visual interactivity 
on top of the audio communication channel. In such 
an implementation, real-time audio and control data 
communications is easily achievable, as the bandwidth 
rcquircmcnt is not significant. 
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APPARATUS AND METHOD FOR REMOTE TROUBLESHOOTING 
Field of invention 

This inveation relates to remote troubleshooting systems. In particular, it relates to 
remote troubleshooting sj^tems using video, audio and data communications. 

Background 

There are prior developments in video conferencing systems for industrial process 
monitoring and troubleshooting. The major benefit of such systems is to reduce 
production downtime that results in cosfly lost of revenue. In addition, customer 
siqjport personnel need not be sent to sites if problems can be handled throu^ video 
conferencing sessions, thus resulting in savings in manpower resources. Another benefit 
is when site service technicians are perfonning the troubleshooting with tiie siqiport of 
a distant customer service expert, these technicians gain woddng knowledge for 
handling similar problems in the future. However, most video conferencing proposals 
are usually hardwired and located at predefined areas or rooms within factory buildings 
that may be a distance away fiom machines being repaired. Moving or installing video 
conferencing devices near the machines may not be feasible or practical. Cables linking 
audio and video devices may impede the movements of the service technicians. 

Real-time video conferencing proposals require significant transmission bandwidth, 
thus limiting the choice for the selecting suitable communications channels. The ITU-T 
H-324 Video Conferencing Standard covers both digital video and audio 
communications over telq)hone networks for video conferencing systems. Such 
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systems woik with conventional telephone networks but are not suitable for mobile 
phone and low-speed Internet ^phcations. 

Video conferencing is usually done via two-way video and audio communications. The 
video cortmiunications portion uses most of flie available transmission bandwidth. In 
troubleshooting appHcations, the video conq)onent of video conferencing originating 
fiom fectory premises forms the main point of interest Thus, on&-way video 
communication is usually sufficient for assisting troubleshooting, thereby resulting in 
reducing the video transmission bandwidth. Two-way audio communications occiq)y 
less transmission bandwidth and real-time audio communications are easily achievable. 
In remote troubleshooting, sending rq)etitive still or motion pictures at a slower rate 
can be an option- However, in such ian ^proach, ttiere is no visual interactivity for 
piiq)ointing the area of focus on the images. Furthermore, voice communication over 
the audio channel for identifying and locating the problematic areas can be confusing 
and may cause wrong interpretation. 

There is therefore a need to provide for portable, wearable and hands-ftee remote 
troubleshooting operatioiL There is also a need to address the limitation in video 
transmission bandwidth and provide one-way video communications and visual 
interactivity for implementing an efficient, as well as cost-effective remote 
troubleshooting solution. 



Summary 

Embodiments of the invention involve a method of implementing remote 
troubleshooting whereby a site service technician is assisted by a distant or multiple 
customer service e>q)erts in identifying and locating the problematic areas using video, 
audio and data communications. Such embodiments enable portable, wearable and 
hands-firee remote troubleshooting operation and address the limitation in transmission 
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bandwidth as well as real-time issues. Visual interactivity is added to provide an 
eflScient, as well as cost-effective solution to remote troubleshooting. 

In accordance with a first aspect of the invention, there is provided an apparatus for 
remote troubleshooting enabling video, audio and data communications between a site 
service technician and a distant customer service expert, the £q>paratus including: 

means for the video conmiunications from the site service technician to the 
distant customer service expert, where the site video can be viewed by the site service 
technician and the distant customer service expert simultaneously; 

means for the audio communications between the site service technician and the 
distant customer service expert; 

means for the data conmiunications between the site service technician and the 
distant customer service expect; 

means for enabling visual mteractivity by placing cursor objects onto the video, 
where the cursor objects descriptions are carried in tihie data communications; and 

means for enabling visual communications by placing text and gr^hics onto the 
video, where the text and the gr^hics descriptions are caiiied in the data 
communications. 

In accordance with a second aspect of the invention, there is provided an q>paratus for 
remote troubleshooting enabling video, audio and data communications between a site 
service technician and a distant customer service expert, the apparatus including: 

means for the audio communications between the site service technician and the 
distant customer service expert; 

means for the data communications between the site service technician and the 
distant customer service expert; 

means for carrying video in the data communications, where the video is made 
up of short video clips; 

means for synchronising playback of the short video clips, where the short video 
clips can be viewed by the site service technician and the distant customer service 
expert simultaneously; 
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means for eooabling visual interactivity by placing cursor objects onto the video, 
where the cursor objects descriptions are carried in the data communications; and 

means for enabling visual communications by placing text and gr^bics onto the 
video, where the text and the grsphics descriptions are carried in the data 
communications. 

In accordance with a third aspect of the invention, there is provided an q>paratus for 
remote troubleshooting enabling still images, audio and data communications between 
a site service technician and a distant customer service e^q^ert, the ^paratus including: 

means for the audio conununications between the site service technician and the 
distant customer service expert; 

means for the data communications between the site service technician and the 
distant customer service expert; 

means for carrying still images in the data communications; 

means for sequencing and displaying the still images to the site service 
technician and the distant customer service expert simultaneously; 

means for enabling visual interactivity by placing cursor objects onto the still 
images, where the cursor objects descriptions are carried in the data communications; 
and 

means for enabling visual communications by placing text and graphics onto the 
still images, where the text and the gr^hics descriptions are carried in the data 
conmiunications. 

In accordance with a fourth aspect of the invention, there is provided an aq[>paratus for 
remote troubleshooting enabling video, audio and data communications hetween a site 
service technician and multiple customer service experts, the ^aratus including: 

means for the audio communications between the site service technician and the 
multiple customer service experts; 

means for the data communications between the site service technician and the 
multiple customer service experts; 
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means for carrying video in the data conununications, where the video is made 
up of short video clips; 

means for synchronising playback of the short video clips, where the short video 
cUps can be viewed by the site service technician and the multq)le customer service 
experts simultaneously; 

means for enabling visual interactivity by placing cursor objects onto the video, 
where the cursor objects descriptions are canied in the data communications; and 

means for enabling visual communications by placing text and gr^hics onto the 
video, where the text and the gr^hics descriptions are carried in the data 
communications. 

In accordance with a fifth asfpect of the inveation, there is provided an apparatus for 
remote troubleshooting enabling still images, audio and data conmaunicatioiis between 
a site service technician and multiple customer service experts, the ^aratus including: 

means for the audio communications between the site s^vice technician and the 
multiple customer service experts; 

means for the data communications between the site service technician and the 
multiple customer service e3q)erts; 

means for carrying still images in the data communications; 

means for sequencing and displaying the stiU images to the site service 
technician and the multiple customer service e}q)erts simultaneously, 

means for enabling visual interactivity by placing cursor objects onto the still 
images, where the ciusor objects descriptions are carried in the data conmmnications; 
and 

means for enabling visual communications by placing text and gr^hics onto the 
stiU images, where the text and the gr^hics descriptions are carried in the data 
commurdcations. 

In accordance with a sixth aspect of the invention, there is provided an ^paratus for 
remote troubleshooting, enabling real-time visual interactivity between a site service 
technician and a distant ciistomer service expert, the apparatus includiag: 
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means for generating a site cursor, where the site cursor is controlled by the site 
service technician; 

means for generating a customer service cursor, where the customer service 
cursor is controlled by the distant customer service expert; 

means for embedding the site cursor's information and the customer service 
cursor's information in data transmissions; 

means for overlaying the site cursor and the customer service cursor onto the 
video that is viewed by the site service technician and the distant customer service 
expert simultaneous!}^ and 

means for overlaying text and graphics onto the video that is viewed by the site 
service technician and the distant customer service expert simultaneously. 

Ih accordance with a seventh aspect of the invention, there is provided an apparatus for 
remote troubleshooting, enabling real-time visual interactivity between a site service 
technician and multiple customer service experts, the ^aratus including: 

means for generatiag a site cursor, where the site cursor is controlled by the site 
service technician; 

means for generating a customer service cursors, where the customer service 
cursors are controlled by the customer service experts; 

means for embedding the site cursor's information and the customer service 
cursors' information in data transmissions; 

means for overlaying the site cursor and the customer service cursors onto the 
video that is viewed by the site service technician and the customer service experts 
simultaneously; and 

means for overlaying text and gr^hics onto the video that is viewed by the site 
service technician and the customer service experts simultaneously. 

In accordance with an eighth aspect of the invention, there is provided a method for 
providing synchronised playback of short video clips with visual interactivity for 
remote troubleshooting between a site service technician and a distant customer service 
expert, the method including the steps of: 
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sending video clips jBx)m the site service technician to the distant customer 
service expert; 

sending control data initiating synchronised playback of tiie video cUps, where 
the video clips are viewed by the site service technician and the distant customer 
service expert simultaneously; 

sending site cursor data fiiom the site service technician to the distant customer 
service expert; 

sending customer service cursor data fiom the distant customer service expert to 
the site service technician; 

overlaying site cursor and customer service cursor onto the video clips using the 
site cursor data and the customer service cursor data respectivel)^ 

sending site text and gr^hics data from the site service technician to Qie distant 
customer service expert; 

sending customer service text and graphics data fiiom the distant customer 
service expert to the site service technician; and 

overlaying text and gr^hics onto the video clips using the site text and 
gK5)hics data and the customer service text and gr£5)hics data respectively. 

In accordance with a ninth aspect of the invention, there is provided a method for 
displaying still images with visual interactivity for remote troubleshooting between a 
site service technician and a distant customer service expert, the method including the 
steps of: 

sending stiU images Gxym the site service technician to the distant customer 
service expert; 

sending control data for sequencing still images for display, where the still 
images are viewed by the site service technician and the distant customer service expert 
simultaneously; 

sending site cursor data fiom the site service technician to the distant customer 
service expert; 

sending customer service cursor data &om the distant customer service expert to 
the site service technician; 
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overlaying site cursor and customer service cursor onto the still images using 
the site cursor data and the customer service cursor data respectively; 

sending site text and gr^hics data fiom the site sovice technician to the distant 
customer service expert; 

sending customer service text and graphics data from the distant customer 
sendee expert to the site service technician; and 

overlaying text and gr^hics onto the still images using flie site text and 
gn^hics data and the customer service text and grsphics data respectively. 

Brief description of the drawings 

Figure I illustrates the functional modules of a remote troubleshooting system at a 
troubleshooting site in accordance witti an embodiment of the invention. 

Figure 2 illustrates the fimctional modules of the remote troubleshooting system of 
Figure 1 at the customer service centre. 

Figure 3 provides an illustration on how a site service technician and a customer service 
expert conmiunicate using the remote troubleshooting system of Figure 1 . 

Table 1 provides a definition of a private_data_section. 

Table 2 provides definitions of data_type fields. 

Table 3 provides a definition of a control_data field. 

Table 4 provides definitions of cursor_type fields. 
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Detailed description 

Real-time video conferencing uses significant transmission bandwidth, especially the 
video conunumcations portion- Thus, the main problem to be addressed is to reduce die 
overall transmission bandwidth requirements, and yet provide an efficient means of 
remote troubleshooting. This problem is addressed by providing a remote 
troubleshooting system according to an embodiment of the invention described 
hereinafter. In troubleshooting applications, the video conq)onent of video conferencing 
originating from the factory premises forms the main point of interest. Thus, one-way 
video conMnunication is usually sufficient for assisting troubleshooting, thereby 
resulting in reducing the video transmission bandwidth Still images or video clips are 
sent if the transmission chaimel is not able to acconmiodate real-time video 
transmission. Acknowledgement is sent back when the still imag^ or video clips are 
received. 

Besides the audio channel, data is also transmitted to enable a site service technician 
and a distant customer service expert to conduct efifective troubleshooting sessions. 
Visual interactivity is achieved by means of overlaying cursor objects into the video 
component Two cursor objects, one relating to site service technician and the other 
relating to the customer service expert, are displayed on videos or images. The 
descriptions on the locations as well as the types of objects to be displayed are 
embedded in the data. Besides the cursor objects, text and gr^hics can be added for 
providing visual informatioiL Real-time audio and data communications are easily 
achievable, as the bandwidth requirements are not significant. 

For the identification of problems during troubleshooting, good quahty video is usually 
required and this is not achievable using real-time video transmission at low 
transmission bandwidth. To overcome this limitation, high quahty video chps can be 
transmitted instead and this need not be done in real-time. After a video clip is 
transmitted, a means of synchronisation is incorporated to enable synchronised 
playback of the video clip at the troubleshooting site and the customer service centre. 
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fhus allowing the site service techniciaa and the customer service expert to view 
ideatical video clips simultaneously, Togefha: with the text, grq)hics and ciffsor 
objects, the system allows a basis for further troubleshooting sessions and discussions 
with the aid of video and visual interactivity in addition to the audio communications 
channel. 

Operation of the svstem 

During a remote troubleshooting session, the site service technician communicates with 
the distant customer service expert using video, audio and data conmiunications over a 
suitable transmission channel such as the hitemet, a telephone line, or a hand-phone or 
other wireless systems. The site smdce technician cq)tures a video of the poiot of 
interest and transmits the video shot to tiie distant customer service expert Still images 
or video cHps are sent if the transmission channel is not able to accorumodate real-time 
video transmission. When the video or still image is being played, the site technician 
highlights the subject in focus using a pouit^ device. The pointer device overlays a 
cursor object onto the video. At customer service centre, the cursor object that is 
created by the site service technician also shears on the video displayed to the 
customer service expert The customer service expert can look into the problem relying 
on the video that is transmitted, the cursor object overlaid onto tihe video, as well as the 
audio communication established with the site service technidaiL The customer service 
expert in turn can request to zoom into certain areas by hi^iKghting the desired portion 
of the video using another cursor object In re^onse to the request, the site technician 
zooms in and cq)tures the video and the troubleshooting process continues as such. 
Thus there are two cursor objects overlaying the video, one being controlled by the site 
service technician and the other by the customer service expert The movements and 
shape of the cursor objects are displayed on the video viewed concurrently by the site 
service technician and the customer service expert In addition to the cursor objects, 
text and grq)hics can be added for visual communications. Thus, visual interactivity is 
achieved by means of overlaying text, gjcapldcs and cursor objects into tihe video, 
assisting in highlighting the area of focus or pinpointing the problematic areas. 
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Description of the system 

Functional modules of the remote troubleshooting system in according to an 
embodiment of the invention for use at the troubleshooting site are shown in Figure 1. 
A pointer device 100 includes a trackball for providing positioning control, as well as 
left and right buttons for signalling user commands. The pointer device 100 allows the 
site service technician to insert and overlay a troubleshooting site cursor onto the video 
or image that is displayed on an LCD (liquid crystal display) display 1 1 L Positioning 
and user signals 101 from the pointer device 100 are processed by a data processor 106. 
The data processor 106 also receives customer service centre data 102 fiom a data 
decoder 103, A data processor 106 relays cursor data 1 13 conq)rising of information in 
relation to the troubleshooting site cursor and tiie customer service centre cursor to a 
video overlay module 1 14. Data for the troubleshooting site cursor is derived from the 
pointer device 1 00 and data for the customer service centre cursor is derived from the 
data decoder 103. Using such cursor information, the video overlay module 1 14 
generates and overlays the cursor objects onto incoming video signal 120 origuiating 
fiom a video capture module 121. In addition to the cursor information, text and 
graphics can be added for providing visual information. Resultant video output 112 is 
displayed on the LCD display 111. There are two cursor obj ects overlaying onto the 
video, one being controlled by the site service technician at tiie troubleshooting site and 
the other by the distant customer service e?q)ert over at a distant customer s^vice 
centre. The video c^ture module 121 captures video data 1 19 using a troubleshooting 
site camera 118 and relays digital video signals 115 & 120 to the video overlay module 
114 andavideo encoder module 116. The c^tured video data 119 can be stored in 
memory 124 for subsequent retrieval. The video encoder module 1 16 incorporates a 
video compression technique for reducing transmission bandwidth. Conq)ressed video 
data 1 17 is sent to a multiplexer 1 10 that multiplexes data from the video encoder 1 16, 
an audio encoder 122 and a data encoder 108 into packetised data stream 135. The 
packetised data stream 135 is sent to a transmitter 134 for transmission to the distant 
customer service centre. 
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The site service technician can send machine data 105 over to the distant customer 
service expert for checking and validadng purposes. The machine data 105 is iq)loaded 
though a serial port 104 and sent to the data processor 106 which relays the machine 
data 107 to the data encoder module 108. The data encoder module 108 incorporates a 
lossless conq)ression technique. Compressed machine data 109 is sent to the 
multiplexer 110, then to the transmitter 134 for transmission to the distant customer 
service centre. In addition to the cursor information, the distant customer service expert 
can also send a set of machine, document and short AV (audioMdeo) clip data to ttie 
site service technician. A receiver 136 receives received data 137 fiom the customer 
service centre and transfers tfie received data 137 to a demultiplexer 133. After de- 
multiplexing, the set of machine, document and the short AV clip data, as weU as 
cursor data 138, are sent to the data decoder 103. The deconD5)ressed data 102 is 
transferred to the data processor 106, which determines the subsequent course of action 
for the data 102. Document data can be processed, routed to the video overlay module 
114 for displaying on the LCD display 111 while the machine data 105 is routed to the 
serial port 1 04. The short AV clip data is archived in a memory 124 for subsequent 
synchronised playback. 

The audio section of the system includes a headset consisting of a microphone 125 and 
a heat^hone 127. An audio encoder 122 encodes a voice signal 126 fiom the 
naicrophone 125 into compressed audio data packets 123. Compression is performed to 
reduce the audio transmission bandwidth. The packetised compressed audio data 123 is 
sent to the multiplexer 1 10 for transmission. The receiver 136 receives audio data fiom 
the distant customer service ^eit. After de-multiplexing, audio data 132 is sent to an 
audio decoder 131. Audio output 130 is mix with the voice signal 126 in an audio 
mixer 129 and the resultant audio output 128 is sent to the headphone 127. 

Functional modules of the remote troubleshooting system at the customer service centre 
are shown in Figure 2. A pointer device 205 generates positioning and user signals 204 
that are sent to a data processor 209. The data processor 209 also receives 
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troubleshooting site data 208 ftom a data decoder 207. The data processor 209 relays 
cursor data 215 comprising of infoimation relating to the troubleshooting site cursor 
and the customer service centre cursor to a video overlay module 216. Data relating to 
customer service cursor is derived firom the pointer device 205 and data relating to the 
troubleshooting site cursor data is derived fix>m the cursor information embedded in 
troubleshooting site data 208 provided by the data decoder 207. Using such cursor 
information, the video overlay module 216 generates and overlays the cursor objects 
onto a video signal 214 provided by a video decoder 213 which decodes a digital video 
sigoal 212 provided by a demultiplexer 200. The digital video data 212 can be stored in 
a memory 219 for subsequent retrieval. The resultant video output 217 is displayed on a 
video display 218- 

Besides the cursor information, the customer service expert can also send a set of 
machine, document and short AV clip data 210 through a serial port 211. The set of 
machine, document and the short AV cUp data 2 1 0, as well as customer service cursor 
data 203, are sent to a data encoder 202, and subsequentiy encoded data 201 is routed to 
a multiplexer 226. Multiplexed output 234 is scat to a transmitter 233 for transmission 
to the site service technician. On the other hand, a receiver 231 receives the 
troubleshooting site data and received data 232 is transferred to a demultiplexer 200. 
After de-multq)lexing, data relating to the site cursor, as well as site machine, document 
and short AV clip data 206 are sent to the data decoder 207. The deconqiressed data 
208 is transferred to the data processor 209. Document data can be processed, routed to 
the video overlay module 216 for displaying on the disfplay 218 while the machine data 
210 is routed to the serial port 21 1 . The short AV cUp data is archived in memory 219 
for subsequent synchronised playback. 

The audio section of the system includes a headset consisting of a microphone 230 and 
a headphone 225. An audio encoder 228 encodes a voice signal 229 from the 
microphone 230 into compressed audio data packets 227. The packetised compressed 
audio data 227 is sent to a multiplexer 226 for transmission to the troubleshooting site. 
The receiver 232 receives audio data from the site service technician. After de- 
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multiplexing, audio data 220 is sent to an audio decoder 221 , Audio output 222 is mix 
with the voice signal 229 in an audio nuxer 223 and the resultant audio output 224 is 
sent to the heac^hone 225. 

Figure 3 shows the communication set-iq) between the site service technician and the 
customer service expert. At the troubleshooting site, the system includes a wearable site 
console 310. The site console 310 interconnects video input 303 to a video camera 301, 
video output 309 to a video display unit 302, audio signals 305 to a headset 300, and a 
positional and control signal 306 to a points device 304. The site console 310 is 
constructed j&om the data encoder 108, the data decoder 103, the data processor 106, 
the video overlay 1 14, the video encoder 1 16, the video capture 121, the memory 124, 
the audio encoder 122, tiie audio mixer 129, the audio decoder 131, the multiplexer 1 10 
and the demultiplexer 133 functional modules as shown in Figure 1. The site console 
310 relays video data, audio data, as well as cursor information, machine or document 
data 311 to a site communicator 312. The site communicator 312 is formed fiom the 
transmitter 134 and the receiver 136 functional modules as shown in Figure 1. The site 
communicator 312 provides communication of digital data between the troubleshooting 
site and the customer service centre. 

Over at the customer service centre;, the system includes a customs service console 
315, interconnecting a video signal 320 to a video display unit 321, audio signals 316 to 
aheadset 317, and a positional and control signal 318 to apointer device 319. The 
customs: service console 315 is constructed £bom the data encoder 202, the data 
decoder 207, the data processor 209, tiie video overlay 216, the video decoder 213, the 
memory 219, the audio encoder 228, the audio mixer 223, the audio decoder 221, the 
multiplexer 226 and the demultiplexer 200 ftmctional modules as sbown in Figure 2. 
The site console 315 relays audio data as well as cursor information, machine, 
document or AV clips data 314 to a customer service cormnunicator 313. The customer 
service communicator 313 is formed ftom the transmitter 233 and the receiver 231 
functional modules as shown in Figure 2. 
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Two cxirsor objects, one relating to the site service technician and the other relating to 
the customer service expert, are displayed on the video or images. At the 
troubleshooting site, the site service technician views the video with the 
troubleshooting site cursor 307 and the customer service centre cursor 308. Over at the 
customer service centre, the customer service expert views the same video with the 
troubleshooting site cursor 322 and the customer service centre cursor 323. The 
descriptions on the locations as well as the types of cursor objects to be displayed are 
embedded in the transmission of the digital data between the troubleshooting site and 
the customer service centre. Data communication for real-time audio and cursor easily 
achievable, as the bandwidth requirement is not significant The positions and 
movements of the cursors shearing over tibie displays at troubleshooting site and the 
customer service centre are in synchronisation. The viewer can toggle the cursors on or 
off. 

hi the embodiment, visual interactivity is achieved using cursor information embedded 
in data transmission in real-time. Machine data, document and short AV clip data can 
also be transmitted but need not be done in real-time. Infoimation relating to the cursor, 
machine, document and short AV cUp data is described in the private data section table 
as shown in Table 1. The semantic definitions are: 

• data_id — A 1 3-bit unique identification for private data section carrying cursor, 
machine and document data. This is used for filtering the private data section from 
the data packets. 

• datajtype - This 3-bit field identifies the type of data that follows. The description 
of the datatype is tabulated in Table 2. 

• controljdata - This 32-bit field contains the control data. The description of the 
control_data is tabulated in Table 3. 

• cursor_type — This 4-bit field describe the type of cursor. The description of the 
cursor jtype is tabulated in Table 4. 

• cursor_size — This 4-bit field indicates the preferred size of the cursor. 

• cursor_style — This 4-bit field indicates the preferred style of the cursor. 
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• x_axis — This 10-bit field contains the horizontal co-ordinates of the cursor. 

• y axis — This 10-bit field contains the vertical co-ordinates of the cursor, 

• x_size — This 10-bit field contains the horizontal size of the rectangular cursor box. 

• y_size — This 1 0-bit field contains the vertical size of the rectangular cursor box. 

• radius ~ This 8-bit field contains the radius of the circular cursor box. 

• text_data_length - This 12-bit field specifies the length of bytes of the following 
text data descriptiorL 

• text_data — This 8-bit field contains the text data. 

• machine_id — A 1 6-bit unique identification for the machine model. 

• machine_data_type — This 6-bit field identifies ttie machine data type. 

• section_number - This is a 16-bit field that gives the number of this 
private_data_sectioiL The private data are retrieved starting fiom section 0 to the 
last_section_number. The numbers are running sequentially. 

• last_section_jmmber — This 1 6-bit field specifies the last section number of the 
private_data_sectioiL 

• machiaejdata__length — This 12-bit field specifies tiie length of bytes of the 
following machine data description. 

• machine_data — This 8-bit field contains flie machine data. 

• document type - This 6-bit field identifies the document type. 

• document_data_length — This 12-bit field specifies the length of bytes of the 
following document data description. 

• document_data — This 8-bit field contains the document data. 

• AVjtype — This 6-bit field identifies the AV data type. 

• AV_version_number — This 8-bit field contains ttie version number of flie AV data. 
It provides an identification tag for retrieval of the archived short video clip or stiU 
images for synchronised playback. 

• AV_data_length - This 12-bit field specifies the length of bytes of the following 
AV data description- 

• AV__data - This 8-bit field contains the short AV cUp data. 
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• CRC — This 32-bit field contains CRC value. It can be used to check the correctness 
of the data in this section. 

For the identification of problems during remote troubleshooting, good quality video is 
usually required. If this is not achievable using real-time video transmission, video clips 
can be transmitted instead and this need not be done in real-time. The short video clips 
or still images can be delivered using the private_data_section with data_type value 
equals S. After a video is transmitted, a means of synchronisation is incorporated to 
enable synchronise playback of the video at the troubleshooting site and the customer 
service cmtre, thus allowing the site service technician and the customer service &xpest 
to view identical videos simultaneously. The synchronisation information is ^bedded 
in the control_data carried in tfie private_data_sectioa Thus, together with the cursor 
objects, the system allows a basis for fiirttier troubleshooting sessions and discussions 
with the aid of video and visual interactivity on top of the audio communication 
channel. Multiple commxmication stations can be set iq;> for communication involving 
three or more parties. 

The system provides an enhancement to existing methods of industrial process 
monitoring and troubleshooting. The major baiefit of such a system is to provide a 
cost-effective remote troubleshooting solution, enabling video, audio and data 
commimications between tiie site service technician and the distant customer service 
expert 
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Claims 

1 . An ^paratus for remote troubleshooting enabling video, audio and data 
communications between a site service technician and a distant customer service expert, 
said apparatus including: 

means for said video conomunications firom said site service technician to said 
distant customer service expert, where the site video can be viewed by said site service 
technician and said distant customer service expert sinaultaneously; 

means for said audio communications between said site service technician and 
said distant customer service expert; 

means for said data conununications between said site service technician and 
said distant customer service e}q>ert; 

means for enabling visual interactivity by placing cursor objects onto said video, 
where said cursor objects descriptions are carried in said data communications; and 

means for enabling visual communications by placing text and gr£^hics onto 
said video, where said text and said gr^hics descriptions are carried in said data 
communications. 

2. An ^paratus for remote troubleshooting enabling video, audio and data 
communications between a site service technician and a distant customer service expert, 
said q)paratus including: 

means for said audio communications between said site service technician and 
said distant customer service expett; 

means for said data communications between said site service technician and 
said distant customer service expert; 

means for carrying video in said data conmiunications, where said video is made 
up of short video chps; 

means for synchronising playback of said short video cUps, where said short 
video clips can be viewed by said site service technician and said distant customer 
service expert simultaneously; 
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means for enabling visual interactivity by placing cursor objects onto said video, 
where said cursor objects descriptions are carded in said data communications; and 

means for enabling visual communications by placing text and gr^hics onto 
said video, where said text and said graphics descriptions are carried in said data 
communications. 

3. An jqpparatus for remote troubleshooting enabling still images, audio and data 
communications between a site service technician and a distant customer service expert, 
said ^aratus including: 

meam for said audio communications between said site service technician and 
said distant customer service expert; 

means for said data communications between said site service technician and 
said distant customer service expert; 

means for carrying still images in said data communications; 

means for sequencing and displaying said still images to said site service 
technician and said distant customer service expect simultaneously; 

means for enabling visual interactivity by placing cursor objects onto said still 
images, where said cursor objects descriptions are carried in said data communications; 
and 

means for enabling visual communications by placing text and graphics onto 
said still images, where said text and said grq)hics descriptions are carried in said data 
communications. 

4. An jqjparatus for remote troubleshooting enabling video, audio and data 
connnunications between a site service technician and multiple customer service 
experts, said ^paratus including: 

means for said audio communications between said site service technician and 
said multiple customer service experts; 

means for said data communications between said site service technician and 
said multiple customer service experts; 
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means for carrying video in said data conununications, where said video is made 
up of short video clips; 

means for synchronistag playback of said short video clips, where said short 
video clips can be viewed by said site service technician and said multiple customer 
service experts simultaneously; 

means for enabling visual interactivity by placing cursor objects onto said video, 
where said cursor objects descriptions are carried in said data communications; and 

means for enabhng visual conmiunications by placing text and grs^hics onto 
said video, where said text and said graphics descriptions are carried in said data 
communications. 

5. An apparatus for remote troubleshooting enabling still images, audio and data 
communications between a site service technician and multiple customer service 
oqperts, said ^aratus including: 

means for said audio communications between said site service technician and 
said multiple customer service experts; 

means for said data communications between said site service technician and 
said multiple customer service experts; 

means for carrying still images in said data communications; 

means for sequencing and displaying said still images to said site service 
technician and said multiple customer service experts simultaneously 

means for enabling visual interactivity by placing cursor objects onto said still 
images, where said cursor objects descriptions are carried in said data conomunications; 
and 

means for enabling visual communications by placing text and gr^hics onto 
said still images, where said text and said graphics descriptions are carried in said data 
communications. 

6. An apparatus for remote troubleshooting, enabling real-time visual interactivity 
between a site service technician and a distant customer service expert, said apparatus 
including: 
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meaias for generating a site cursor, where said site cursor is controlled by said 
site service technician; 

means for generating a customer service cursor, where said customer service 
ciffsor is controlled by said distant customer service expert; 

means for embedding said site cursor's iofonnation and said customer service 
cursor's information in data transmissions; 

means for overlaying said site cursor and said customer service cursor onto the 
video that is viewed by said site service technician and said distant customer service 
expert simultaneously; and 

means for overlaying text and grq)hics onto said video that is viewed by said 
site service technician and said distant customer service expert simultaneously. 

7. An ^aratus for remote troubleshooting, enabling real-time visual interactivity 
between a site service technician and multiple customer service experts, said q^paratos 
including: 

means for generating a site cursor, where said site cursor is controlled by said 
site service technician; 

means for generating a customer service cursors, where said customer service 
cursors are controlled by said customer service experts; . 

means for embedding said site cursor's information and said customer service 
cursors' information in data transmissions; 

means for overlaying said site cursor and said customer service cursors onto the 
video that is viewed by said site service technician and said customer service experts 
simultaneously; and 

means for overlaying text and gr^hics onto said video that is viewed by said 
site service technician and said customer service experts simultaneously. 

8. A method for providing synchronised playback of short video cUps with visual 
interactivity for remote troubleshooting between a site service technician and a distant 
customer service expert, said method including the steps of: 
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sending video clips from said site service technician to said distant customer 
service expert; 

sending control data initiating synchronised playback of said video clips, where 
said video clips are viewed by said site service technician and said distant customer 
service expert simultaneously; 

sending site cursor data from said site service technician to said distant customer 
service expert; 

sending customer service cursor data from said distant customer service expert 
to said site service technician; 

overlaying site cursor and customer service cursor onto said video clips using 
said site cursor data and said customer service cursor data respectively; 

sending site text and grq)hics data from said site service technician to said 
distant customer service expert; 

sending customer service text and graphics data from said distant customer 
service expert to said site service technician; and 

overlaying teict and gr^hics onto said video cUps using said site text and 
grq)hics data and said customer service text and graphics data respectively. 

9. A method for displaying still images with visual interactivity for remote 
troubleshooting between a site service technician and a distant customer service expert, 
said method including the steps of: 

sending still images from said site service technician to said distant customer 
service expert; 

sending control data for sequencing still images for display, where said still 
images are viewed by said site service technician and said distant customer service 
OTpert simultaneously, 

sending site cursor data from said site service technician to said distant customer 
service expert; 

sending customer service cursor data from said distant customer service expert 
to said site service technician; 
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overlaying site cursor and customer service cursor onto said still images using 
said site cursor data and said customer service cursor data respectively; 

sending site text and gr^hics data fix>m said site service technician to said 
distant customer service expert; 

sending customer service text and grq>hics data from said distant customer 
service expert to said site service technician; and 

overlaying text and grs^hics onto said still images using said site text and 
graphics data and said customer service text and graphics data respectively. 
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Figure 1 : Functional blocks of flie rcmote troubleshooting system at flie troubleshooting site. 
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Figure 2: Functional blocks of the remote troubleshooting system at the customer service 
centre. 
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Figure 3: Communication between the site service technician and the customer service expert 
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Table 1: private data section -4/5- 



Syntax 


No. of bits 






clata_ia 


ID 


data_Jype 


% 
J 


11 (data_type = ==0) { 




control_data 

J 

else 11 (datajype = = 1) ( 






cursortype 


A 


cursor_size 


4 


cursor_style 


4 


x_axis 


lU 


y_axis 


10 


if (cursoMype = = 1) { 




reserved 


4 


x_size 


in 
iU 


y_size 

} 


10 


else if (cursoMype = = 2){ 




radius 

} 

} 


8 




else 11 (aata_1ype = = 2) { 




reserved 


4 


text_data_lengtii 


12 


for (1=0; i<lext_data_lengfl»; 1++) { 




text_data 


o 

8 


} 




} 




else 11 (aata_Jype = = 3) { 




niachiiie_id 


JLO 


reserved 


2 


machinejdata^type 


D 


section_number 


16 


last_section_nuinber 


ID 


reserved 


4 


inacliine_data_length 


1 *> 


for (1=0; i<inachme_aata_lengtD; i-H-) { 




maclime_aata 

} 


O 

o 


} 

else if (data__type — = 4) { 




reserved 


2 


docmnent_type 


o 


section_number 


16 


last_section_number 


10 


reserved 


4 


docuinent_data_lengtii 


12 


for (i=0; i<document_data_length; i-H-) { 




document data 

} 


8 


} 

else if (data type = = 5) { 
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reserved 


2 


AVjype 


6 


sectionjQinnber 


16 


last_section_number 


16 


AV_version_nuinber 


8 


reserved 


4 


AV_data_length 


12 


for (i=0; i<AV_data lengfli; i-hf) { 




AV data 

} 


8 


} 

CRC 32 


32 


I 





Table 2: data type definitions 



Data Type 


data_type 
value 


Desa^tions 


Controls 


0 


Control information. 


Cursor 


I 


Cursor positional and control attributes data. 


Text/Graphics 


2 


Text or graphics data for delivering visual messages. 


Machine 


3 


Machine data such as numerical control or system information. 


Document 


4 


Document files. 


AV 


5 


Batch audio & video data. For non real-time transmission. 


Reserved 


6-7 


Not used in current inoEplementation. 



Table 3: control data definitions 



Syntax 


No. of bits 


control_dataO { 




control_command 


8 


control_parameters 


24 


} 





Table 4: cursor t 


ype definitions 


Cursor Type 


cursor_type 
value 


Descriptions 


Arrow 


0 


Pointer for identifedng the point of interest 


Rectangle 


1 


A rectangular box for indicating the area of interest 


Circle 


2 


A circular box for indicating the area of interest 


Reserved 


3-lS 


Not used in current implementation. 



